Linear Discriminant Analysis: New Formulations and Overfit Analysis

In this paper, we will present a unified view for LDA. We will (1) emphasize that standard LDA solutions are not unique, (2) propose several new LDA formulations: St-orthonormal LDA, Sw-orthonormal LDA and orthogonal LDA which have unique solutions, and (3) show that with St-orthonormal LDA and Sw-orthonormal LDA formulations, solutions to all four major LDA objective functions are identical. Furthermore, we perform an indepth analysis to show that the LDA sometimes performs poorly due to over-fitting, i.e., it picks up PCA dimensions with small eigenvalues. From this analysis, we propose a stable LDA which uses PCA first to reduce to a small PCA subspace and do LDA in the subspace. Introduction Linear discriminant analysis (LDA)(Fisher 1936) is widely used classification method, especially in applications where the data dimension is large, such as in computer vision(Turk and Pentland 1991; Belhumeur, Hespanha, and Kriengman 1997) where data objects are images with typically 100 dimensions. Since it is invented in late 1940’s, there is a large number of studies on LDA methodology, among them Fukunaga’s book(Fukunaga 1990) is the most authoritative. Since 1990, there are many developments, such as uncorrelated LDA(Jin et al. 2001), orthogonal LDA, (Ye and Xiong 2006), null-space LDA(Chen et al. 2000), and a host of other methods such as generalized SVD (Park and Howland 2004) for LDA, 2DLDA (Ye et al. 2004; Luo, Ding, and Huang 2009) , etc. For the simple formulation of LDA of Eq. (1), it is a bit surprising to have this large number of varieties. In this paper, we undertake a different route. Instead of developing newer methods, we ask a few fundamental questions about LDA. Given the fact that there are so many LDA varieties, a natural question is: is the LDA solution unique? A related question: is the LDA solution global solution? Consulting on Fukunaga’s book and other books(Duda, Hart, and Stork 2000; Hastie, Tibshirani, and Friedman 2001), and reading recent papers, these questions were not addressed (or not emphasized at least), to the best of our knowledge. Copyright c © 2011, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. Our investigation of this neglected area of LDA uncover a large number of new results regarding uniqueness, normalization, global solutions. We also investigate the LDA overfitting problem. Our experiments on real life datasets show that LDA often overfits by incorporating PCA (principal component analysis) dimensions with small eigenvalues which causes poor performance. Our results suggest several new approaches to improve the performance. Outline of New Results We first introduce the LDA formulation and outline the new results. Classic LDA. In LDA, the optimal subspace G = (g1, · · · ,gk) is obtained by optimizing max G J1(G) = Tr GSbG GSwG (1) where the between-class (Sb) and within-class (Sb) scatter matrices are defined as Sb = ∑ k nk(mk −m)(mk −m)T , (2)